find UW star files avoiding hard-coded names #91
Conversation
| if self._files: | ||
| # The files have already been indexed. | ||
| return | ||
| files = sorted(glob.glob(os.path.join(input_dir, |
There was a problem hiding this comment.
A description of the format for input file basenames would be helpful, either in a comment or use a regular expression to pick out fields of interest, which can be named, e.g.
pat = "stars_chunk_(?P<min_htm_id>\d+)_(?P<max_htm_id>\d+).parquet"
Then lines 25-28 below get replaced with
m = re.match(pat, os.path.basename(item))
self._files[(m['min_htm_id'], m['max_htm_id'])] = item
There was a problem hiding this comment.
I dislike regular expressions, but if u really think this is better and generally easier for people to understand (I don't particularly), I can change it.
There was a problem hiding this comment.
I find the regular expression clearer; I don't know whether most people do or not. Adding a comment describing the structure of the basename instead is fine with me.
|
|
||
| return [os.path.join(dirpath, fname) for fname in fnames] | ||
| def find_files(self, pixel, nside=32, res_factor=16): |
There was a problem hiding this comment.
A docstring for this routine would be welcome. Why was 16 chosen as default for res_factor?
There was a problem hiding this comment.
yeah, I'll add a comment. resfactor 16 is obviously the resfactor used by those files.
There was a problem hiding this comment.
What does resfactor have to do with HTM depth (apparently 20), if anything?
Healpixels with nside=32 are much bigger than HTM depth 20 regions (or depth 16 regions, for that matter). How did you verify that nside*16 gives a sufficiently fine grid?
There was a problem hiding this comment.
Another way to do this (which seems safer to me) would be to define a minimum circle around the healpixel, then call HtmIndexer.getShardIds
…or and looking up HTM indices
jchiang87
force-pushed
the
u/jchiang/find_UW_star_files
branch
from
1cc74ee
to
4853818
Compare
This samples the desired healpixel, computing the corresponding HTM index at each point, and adds that corresponding file to the list.